DCS Comprehensive Health Plan

Inter-Rater Reliability Testing

Statement/Purpose

Inter-Rater Reliability (IRR) testing is utilized as a mechanism to monitor and evaluate comprehension of medical criteria, consistent decisions and to ensure accurate and consistent application of the criteria among all staff involved with the process.

Inter-Rater Reliability testing is conducted in an effort to:

• Minimize variation in the application of clinical guidelines;

• Evaluate staff ability to identify potentially avoidable utilization;

• Evaluate staff ability to identify quality of care concerns;

• Evaluate staff ability to triage, systematically examine and finalize quality of care concerns;

• Target specific areas most in need of improvement;

• Identify staff needing additional training; and

• Verify the collection of data utilized to measure performance is consistent and comparable among all staff involved with the process. e.g. to ensure the integrity and validity of medical record abstraction.

Definitions

• Inter-Rater Reliability (IRR) Testing: A process to assess whether participants in a process interpret and implement criteria in a consistent and comparable manner. This process is utilized to identify inconsistencies and provide remediation to ensure congruence in outcomes of the task. This process may include the interpretation of Evidence-based criteria application, navigation and knowledge of agency policies and procedures, criteria used in data collection and other applicable activities.

• HEDIS: Healthcare Effectiveness Data and Information Set is a tool used by more than 90 percent of America's health plans to measure performance on important dimensions of care and service.

• NCQA: is a private, 501 (c)(3) not-for profit organization in the United States dedicated to improving health care quality. It does so through the administration of evidence-based standards, measures, programs and accreditation.

Policy

Inter-Rater Reliability (IRR) testing is completed as needed by DCS CHP and/or its contracted MCO staff who are involved in the collection of clinical data related to performance measures, in clinical decision making processes and in any other processes as needed.

DCS CHP and/or its contracted MCO ensure that staff conducting utilization review tasks as well as staff who abstract and report HEDIS hybrid data are able to interpret criteria and collect information consistently and uniformly.

DCS CHP and/or its contracted MCO trains and tests staff responsible for these activities prior to them performing the duties and responsibilities of the position.

Procedure

The process is two-fold and includes:

• Training on the criteria or performance measure guidelines as well as process for the activity;,

• Inter-rater reliability training and testing for consistency and accuracy of the process involved ; and

• Quality control oversight to validate accuracy of inter-rater activity.

The MCO performs this process in alignment with their policy on Inter Rater Reliability, that DCS CHP reviews Ad Hoc to be in alignment with DCS CHP policy and AHCCCS policy.

Testing

IRR tests occurs for all staff involved in clinical authorization decisions, at least annually.

Instructions are provided to staff for accessing and completing IRR testing, including time frames for completion.

IRR testing includes individual case scenarios, as well as cases developed to review highly specialized criteria such as behavioral health or transplant related areas.

Once testing is complete scores are tabulated to determine clinician performance.

Aggregate data is collected for presentation to the Chief Medical Officer and to the Quality Management/Performance Improvement (QM/PI) Committee.

Testing Frequency

IRR testing is conducted annually at a minimum, or more often to ensure process fidelity and consistency in decision making.

Testing for newly hired staff is conducted upon completion of training and annually thereafter.

Compliance

Clinical staff should achieve or exceed 90% consistency with determinations. Individual education plans are developed for those staff who do not meet the expected rate.

Results and Reporting

Individual IRR testing results are kept confidential.

Collective results are reported to DCS CHP and summarized for discussion at the Medical Management (MM) and/or Quality Management/Performance Improvement (QM/PI) committee meetings to identify opportunities for improvement and additional training needs when trends are identified.

DCS CHP’s contracted Managed Care Organization (MCO) reports results of IRR testing of the MCO clinical staff to DCS CHP. Any DCS CHP staff that participate in the UM decision making are also included in the MCO IRR process.